A toponym-based dual vector for topical relevance calculation in focused spatial crawling

Focused crawler is a Web crawler that tries to download only pages that are relevant to a given topic of interest (Siemiński 2009, Almpanidis 2011). That is to say, it is necessary for focused crawler to calculate relevance between pages and specific topic (Rungsawang, 2005). Recently, the specific topic involving spatial information especially toponyms such as the topic about the Diaoyu Islands conflict between China and Japan has become much more, because there are about 18 percent webpages describing localization information and 70 percent webpages containing geographic information webpages in the worldwide webpages (Zhou et al. 2005, Hill 2009). For the given topic involving toponyms, it means that focused crawler should ensure the downloaded pages are not only relevant with common topic and also relevant with the toponyms, which requires more accurate topical relevance calculation. At present, most researchers adopt Vector Space Model (VSM) (Fox 1984, Batsakis et al. 2009) or improved VSM (Yang et al. 2010, Li 2008) to calculate the topical relevance. In these methods the given topic and document are regarded as one keywords vector, which means that toponyms are consider as common topic. Considering toponyms as common topic may reduce the crawling accuracy, because a toponym can involve different common topics and a common topic also can relate to many toponyms. To solve the problem, we make a difference between toponyms and common keywords according to the idea of Geographic Information Retrieval (Purves 2007) and develop a toponym-based dual vector for topical relevance calculation.